From Bench to Bedside With Large Language Models: AJR Expert Panel Narrative Review.

Large language models (LLMs) hold immense potential to revolutionize radiology. However, their integration into practice requires careful consideration. Artificial intelligence (AI) chatbots and general-purpose LLMs have potential pitfalls related to privacy, transparency, and accuracy, limiting their current clinical readiness. Thus, LLM-based tools must be optimized for radiology practice to overcome these limitations. While research and validation for radiology applications remain in their infancy, commercial products incorporating LLMs are becoming available alongside promises of transforming practice. To help radiologists navigate this landscape, this AJR Expert Panel Narrative Review provides a multidimensional perspective on LLMs, encompassing considerations from bench (development and optimization) to bedside (use in practice). At present, LLMs are not autonomous entities that can replace expert decision-making, and radiologists remain responsible for the content of their reports. Patient-facing tools, particularly medical AI chatbots, require additional guardrails to ensure safety and prevent misuse. Still, if responsibly implemented, LLMs are well-positioned to transform efficiency and quality in radiology. Radiologists must be well-informed and proactively involved in guiding the implementation of LLMs in practice to mitigate risks and maximize benefits to patient care.

Evaluating the Robustness of a Deep Learning Bone Age Algorithm to Clinical Image Variation Using Computational Stress Testing.

Purpose To evaluate the robustness of an award-winning bone age deep learning (DL) model to extensive variations in image appearance. Materials and Methods In December 2021, the DL bone age model that won the 2017 RSNA Pediatric Bone Age Challenge was retrospectively evaluated using the RSNA validation set (1425 pediatric hand radiographs; internal test set in this study) and the Digital Hand Atlas (DHA) (1202 pediatric hand radiographs; external test set). Each test image underwent seven types of transformations (rotations, flips, brightness, contrast, inversion, laterality marker, and resolution) to represent a range of image appearances, many of which simulate real-world variations. Computational "stress tests" were performed by comparing the model's predictions on baseline and transformed images. Mean absolute differences (MADs) of predicted bone ages compared with radiologist-determined ground truth on baseline versus transformed images were compared using Wilcoxon signed rank tests. The proportion of clinically significant errors (CSEs) was compared using McNemar tests. Results There was no evidence of a difference in MAD of the model on the two baseline test sets (RSNA = 6.8 months, DHA = 6.9 months; P = .05), indicating good model generalization to external data. Except for the RSNA dataset images with an appended radiologic laterality marker (P = .86), there were significant differences in MAD for both the DHA and RSNA datasets among other transformation groups (rotations, flips, brightness, contrast, inversion, and resolution). There were significant differences in proportion of CSEs for 57% of the image transformations (19 of 33) performed on the DHA dataset. Conclusion Although an award-winning pediatric bone age DL model generalized well to curated external images, it had inconsistent predictions on images that had undergone simple transformations reflective of several real-world variations in image appearance. Keywords: Pediatrics, Hand, Convolutional Neural Network, Radiography Supplemental material is available for this article. © RSNA, 2024 See also commentary by Faghani and Erickson in this issue.

The clinician-AI interface: intended use and explainability in FDA-cleared AI devices for medical image interpretation.

As applications of AI in medicine continue to expand, there is an increasing focus on integration into clinical practice. An underappreciated aspect of this clinical translation is where the AI fits into the clinical workflow, and in turn, the outputs generated by the AI to facilitate clinician interaction in this workflow. For instance, in the canonical use case of AI for medical image interpretation, the AI could prioritize cases before clinician review or even autonomously interpret the images without clinician review. A related aspect is explainability - does the AI generate outputs to help explain its predictions to clinicians? While many clinical AI workflows and explainability techniques have been proposed, a summative assessment of the current scope in clinical practice is lacking. Here, we evaluate the current state of FDA-cleared AI devices for medical image interpretation assistance in terms of intended clinical use, outputs generated, and types of explainability offered. We create a curated database focused on these aspects of the clinician-AI interface, where we find a high frequency of "triage" devices, notable variability in output characteristics across products, and often limited explainability of AI predictions. Altogether, we aim to increase transparency of the current landscape of the clinician-AI interface and highlight the need to rigorously assess which strategies ultimately lead to the best clinical outcomes.

Evaluating the Use of ChatGPT to Accurately Simplify Patient-centered Information about Breast Cancer Prevention and Screening.

Purpose To evaluate the use of ChatGPT as a tool to simplify answers to common questions about breast cancer prevention and screening. Materials and Methods In this retrospective, exploratory study, ChatGPT was requested to simplify responses to 25 questions about breast cancer to a sixth-grade reading level in March and August 2023. Simplified responses were evaluated for clinical appropriateness. All original and simplified responses were assessed for reading ease on the Flesch Reading Ease Index and for readability on five scales: Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, Automated Readability Index, and the Simple Measure of Gobbledygook (ie, SMOG) Index. Mean reading ease, readability, and word count were compared between original and simplified responses using paired t tests. McNemar test was used to compare the proportion of responses with adequate reading ease (score of 60 or greater) and readability (sixth-grade level). Results ChatGPT improved mean reading ease (original responses, 46 vs simplified responses, 70; P < .001) and readability (original, grade 13 vs simplified, grade 8.9; P < .001) and decreased word count (original, 193 vs simplified, 173; P < .001). Ninety-two percent (23 of 25) of simplified responses were considered clinically appropriate. All 25 (100%) simplified responses met criteria for adequate reading ease, compared with only two of 25 original responses (P < .001). Two of the 25 simplified responses (8%) met criteria for adequate readability. Conclusion ChatGPT simplified answers to common breast cancer screening and prevention questions by improving the readability by four grade levels, though the potential to produce incorrect information necessitates physician oversight when using this tool. Keywords: Mammography, Screening, Informatics, Breast, Education, Health Policy and Practice, Oncology, Technology Assessment Supplemental material is available for this article. © RSNA, 2023.

Examination-Level Supervision for Deep Learning-based Intracranial Hemorrhage Detection on Head CT Scans.

Purpose To compare the effectiveness of weak supervision (ie, with examination-level labels only) and strong supervision (ie, with image-level labels) in training deep learning models for detection of intracranial hemorrhage (ICH) on head CT scans. Materials and Methods In this retrospective study, an attention-based convolutional neural network was trained with either local (ie, image level) or global (ie, examination level) binary labels on the Radiological Society of North America (RSNA) 2019 Brain CT Hemorrhage Challenge dataset of 21 736 examinations (8876 [40.8%] ICH) and 752 422 images (107 784 [14.3%] ICH). The CQ500 (436 examinations; 212 [48.6%] ICH) and CT-ICH (75 examinations; 36 [48.0%] ICH) datasets were employed for external testing. Performance in detecting ICH was compared between weak (examination-level labels) and strong (image-level labels) learners as a function of the number of labels available during training. Results On examination-level binary classification, strong and weak learners did not have different area under the receiver operating characteristic curve values on the internal validation split (0.96 vs 0.96; P = .64) and the CQ500 dataset (0.90 vs 0.92; P = .15). Weak learners outperformed strong ones on the CT-ICH dataset (0.95 vs 0.92; P = .03). Weak learners had better section-level ICH detection performance when more than 10 000 labels were available for training (average f1 = 0.73 vs 0.65; P < .001). Weakly supervised models trained on the entire RSNA dataset required 35 times fewer labels than equivalent strong learners. Conclusion Strongly supervised models did not achieve better performance than weakly supervised ones, which could reduce radiologist labor requirements for prospective dataset curation. Keywords: CT, Head/Neck, Brain/Brain Stem, Hemorrhage Supplemental material is available for this article. © RSNA, 2023 See also commentary by Wahid and Fuentes in this issue.

Appropriateness of Information Provided by ChatGPT Regarding Breast Pathologic Diagnoses.

A multidisciplinary physician team rated information provided by ChatGPT regarding breast pathologic diagnoses. ChatGPT responses were mostly appropriate regarding accuracy, consistency, definitions provided, and clinical significance conveyed. Responses were scored lower in terms of management recommendations provided, primarily related to low agreement with recommendations for high-risk lesions.

AI-Generated Clinical Summaries Require More Than Accuracy.

This Viewpoint discusses AI-generated clinical summaries and the necessity of transparent development of standards for their safe rollout.

Evaluation of Climate-Aware Metrics Tools for Radiology Informatics and Artificial Intelligence: Toward a Potential Radiology Ecolabel.

Radiology is a major contributor to health care's impact on climate change, in part due to its reliance on energy-intensive equipment as well as its growing technological reliance. Delivering modern patient care requires a robust informatics team to move images from the imaging equipment to the workstations and the health care system. Radiology informatics is the field that manages medical imaging IT. This involves the acquisition, storage, retrieval, and use of imaging information in health care to improve access and quality, which includes PACS, cloud services, and artificial intelligence. However, the electricity consumption of computing and the life cycle of various computer components expands the carbon footprint of health care. The authors provide a general framework to understand the environmental impact of clinical radiology informatics, which includes using the international Greenhouse Gas Protocol to draft a definition of scopes of emissions pertinent to radiology informatics, as well as exploring existing tools to measure and account for these emissions. A novel standard ecolabel for radiology informatics tools, such as the Energy Star label for consumer devices or Leadership in Energy and Environmental Design certification for buildings, should be developed to promote awareness and guide radiologists and radiology informatics leaders in making environmentally conscious decisions for their clinical practice. At this critical climate juncture, the radiology community has a unique and pressing obligation to consider our shared environmental responsibility in innovating clinical technology for patient care.

Systematic review of artificial intelligence development and evaluation for MRI diagnosis of knee ligament or meniscus tears.

OBJECTIVE: The purpose of this systematic review was to summarize the results of original research studies evaluating the characteristics and performance of deep learning models for detection of knee ligament and meniscus tears on MRI. MATERIALS AND METHODS: We searched PubMed for studies published as of February 2, 2022 for original studies evaluating development and evaluation of deep learning models for MRI diagnosis of knee ligament or meniscus tears. We summarized study details according to multiple criteria including baseline article details, model creation, deep learning details, and model evaluation. RESULTS: 19 studies were included with radiology departments leading the publications in deep learning development and implementation for detecting knee injuries via MRI. Among the studies, there was a lack of standard reporting and inconsistently described development details. However, all included studies reported consistently high model performance that significantly supplemented human reader performance. CONCLUSION: From our review, we found radiology departments have been leading deep learning development for injury detection on knee MRIs. Although studies inconsistently described DL model development details, all reported high model performance, indicating great promise for DL in knee MRI analysis.

Assessing AI-Powered Patient Education: A Case Study in Radiology.

RATIONALE AND OBJECTIVES: With recent advancements in the power and accessibility of artificial intelligence (AI) Large Language Models (LLMs) patients might increasingly turn to these platforms to answer questions regarding radiologic examinations and procedures, despite valid concerns about the accuracy of information provided. This study aimed to assess the accuracy and completeness of information provided by the Bing Chatbot-a LLM powered by ChatGPT-on patient education for common radiologic exams. MATERIALS AND METHODS: We selected three common radiologic examinations and procedures: computed tomography (CT) abdomen, magnetic resonance imaging (MRI) spine, and bone biopsy. For each, ten questions were tested on the chatbot in two trials using three different chatbot settings. Two reviewers independently assessed the chatbot's responses for accuracy and completeness compared to an accepted online resource, radiologyinfo.org. RESULTS: Of the 360 reviews performed, 336 (93%) were rated "entirely correct" and 24 (7%) were "mostly correct," indicating a high level of reliability. Completeness ratings showed that 65% were "complete" and 35% were "mostly complete." The "More Creative" chatbot setting produced a higher proportion of responses rated "entirely correct" but there were otherwise no significant difference in ratings based on chatbot settings or exam types. The readability level was rated eighth-grade level. CONCLUSION: The Bing Chatbot provided accurate responses answering all or most aspects of the question asked of it, with responses tending to err on the side of caution for nuanced questions. Importantly, no responses were inaccurate or had potential to cause harm or confusion for the user. Thus, LLM chatbots demonstrate potential to enhance patient education in radiology and could be integrated into patient portals for various purposes, including exam preparation and results interpretation.

Economic and Environmental Costs of Cloud Technologies for Medical Imaging and Radiology Artificial Intelligence.

Radiology is on the verge of a technological revolution driven by artificial intelligence (including large language models), which requires robust computing and storage capabilities, often beyond the capacity of current non-cloud-based informatics systems. The cloud presents a potential solution for radiology, and we should weigh its economic and environmental implications. Recently, cloud technologies have become a cost-effective strategy by providing necessary infrastructure while reducing expenditures associated with hardware ownership, maintenance, and upgrades. Simultaneously, given the optimized energy consumption in modern cloud data centers, this transition is expected to reduce the environmental footprint of radiologic operations. The path to cloud integration comes with its own challenges, and radiology informatics leaders must consider elements such as cloud architectural choices, pricing, data security, uptime service agreements, user training and support, and broader interoperability. With the increasing importance of data-driven tools in radiology, understanding and navigating the cloud landscape will be essential for the future of radiology and its various stakeholders.

Coarse Race and Ethnicity Labels Mask Granular Underdiagnosis Disparities in Deep Learning Models for Chest Radiograph Diagnosis.

See also the editorial by Nikolic in this issue.

Applying Artificial Intelligence to Pediatric Chest Imaging: Reliability of Leveraging Adult-Based Artificial Intelligence Models.

OBJECTIVE: The scarcity of artificial intelligence (AI) applications designed for use in pediatric patients can cause a significant health disparity in this vulnerable population. We investigated the performance of an adult-trained algorithm in detecting pneumonia in a pediatric population to explore the viability of leveraging adult-trained algorithms to accelerate pediatric AI research. METHODS: We analyzed a publicly available pediatric chest x-ray dataset using an AI algorithm from TorchXRayVision. A 60% threshold was used to make binary predictions for pneumonia presence. Predictions were compared with dataset labels. Performance measures including true-positive rate, false-positive rate, true-negative rate, false-negative rate, sensitivity, specificity, positive predictive value (PPV), negative predictive value, accuracy, and F1-score were calculated for the complete dataset and bacterial and viral pneumonia subsets. RESULTS: Overall (n = 5,856), the algorithm identified 3,923 cases with pneumonia (67.00%) and 1,933 (33.00%) normal cases. In comparison with the actual image labels, there were 3,411 (58.25%) true-positive cases, 512 (8.74%) false-positive cases, 1,071 (18.29%) true-negative cases, and 862 (14.72%) false-negative cases resulting in 79.83% sensitivity, 67.66% specificity, 86.95% PPV, 55.41% negative predictive value, and 76.54% accuracy and an F1-score of 0.83. Although the performance remained consistent in the bacterial pneumonia group, there was a significant decrease in PPV (69.9%) and F1-score (0.74) in the viral pneumonia group. CONCLUSION: An adult-trained model adequately detected pneumonia in pediatric patients aged 1 to 5 years. Though models trained exclusively on pediatric images performed better, leveraging adult-based algorithms and datasets can expedite pediatric AI research.

Exploring the Clinical Translation of Generative Models Like ChatGPT: Promise and Pitfalls in Radiology, From Patients to Population Health.

Generative artificial intelligence (AI) tools such as GPT-4, and the chatbot interface ChatGPT, show promise for a variety of applications in radiology and health care. However, like other AI tools, ChatGPT has limitations and potential pitfalls that must be considered before adopting it for teaching, clinical practice, and beyond. We summarize five major emerging use cases for ChatGPT and generative AI in radiology across the levels of increasing data complexity, along with pitfalls associated with each. As the use of AI in health care continues to grow, it is crucial for radiologists (and all physicians) to stay informed and ensure the safe translation of these new technologies.

Clinical Applications, Challenges, and Recommendations for Artificial Intelligence in Musculoskeletal and Soft Tissue Ultrasound: AJR Expert Panel Narrative Review.

Artificial intelligence (AI) is increasingly used in clinical practice for musculoskeletal imaging tasks, such as disease diagnosis and image reconstruction. AI applications in musculoskeletal imaging have focused primarily on radiography, CT, and MRI. Although musculoskeletal ultrasound stands to benefit from AI in similar ways, such applications have been relatively underdeveloped. In comparison with other modalities, ultrasound has unique advantages and disadvantages that must be considered in AI algorithm development and clinical translation. Challenges in developing AI for musculoskeletal ultrasound involve both clinical aspects of image acquisition and practical limitations in image processing and annotation. Solutions from other radiology subspecialties (e.g., crowdsourced annotations coordinated by professional societies), along with use cases (most commonly rotator cuff tendon tears and palpable soft tissue masses), can be applied to musculoskeletal ultrasound to help develop AI. To facilitate creation of high-quality imaging datasets for AI model development, focus should be given to increasing uniformity in musculoskeletal ultrasound performance by technologists and radiologists, and to annotation of images for specific anatomic regions. This AJR Expert Panel Narrative Review summarizes available evidence regarding AI's potential utility in musculoskeletal ultrasound, and challenges facing its development. Recommendations for future AI advancement and clinical translation in musculoskeletal ultrasound are discussed.

Use of ChatGPT, GPT-4, and Bard to Improve Readability of ChatGPT's Answers to Common Questions About Lung Cancer and Lung Cancer Screening.

ChatGPT's responses to questions about lung cancer and LCS, although deemed clinically appropriate by cardiothoracic radiologists, were difficult to read. Simplified responses from three LLMs (ChatGPT, GPT-4, and Bard) had improved reading ease and readability (in terms of U.S. grade levels). However, some simplified responses were no longer clinically appropriate.